Chapter 15 

Sums & Asymptotics 



15.1 The Value of an Annuity 

Would you prefer a million dollars today or $50,000 a year for the rest of your life? 
On the one hand, instant gratification is nice. On the other hand, the total dollars 
received at $50K per year is much larger if you live long enough. 

Formally, this is a question about the value of an annuity. An annuity is a finan- 
cial instrument that pays out a fixed amount of money at the beginning of every 
year for some specified number of years. In particular, an n-year, m-payment an- 
nuity pays m dollars at the start of each year for n years. In some cases, n is finite, 
but not always. Examples include lottery payouts, student loans, and home mort- 
gages. There are even Wall Street people who specialize in trading annuities. 

A key question is what an annuity is worth. For example, lotteries often pay 
out jackpots over many years. Intuitively, $50, 000 a year for 20 years ought to be 
worth less than a million dollars right now. If you had all the cash right away, you 
could invest it and begin collecting interest. But what if the choice were between 
$50, 000 a year for 20 years and a half million dollars today? Now it is not clear 
which option is better. 

In order to answer such questions, we need to know what a dollar paid out 
in the future is worth today. To model this, let's assume that money can be in- 
vested at a fixed annual interest rate p. WeTl assume an 8% rate^ for the rest of the 
discussion. 

Here is why the interest rate p matters. Ten dollars invested today at interest 
rate p will become (1+p) • 10 = 10.80 dollars in a year, (1+p)^ • 10 ~ 11.66 dollars 
in two years, and so forth. Looked at another way, ten dollars paid out a year from 
now are only really worth 1/(1 +p) 10 ~ 9.26 dollars today. The reason is that if we 



^U.S. interest rates have dropped steadily for several years, and ordinary bank deposits now earn 
around 1.5%. But just a few years ago the rate was 8%; this rate makes some of our examples a little 
more dramatic. The rate has been as high as 17% in the past thirty years. 

In Japan, the standard interest rate is near zero%, and on a few occasions in the past few years has 
even been slightly negative. It's a mystery why the Japanese populace keeps any money in their banks. 
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had the $9.26 today, we could invest it and would have $10.00 in a year an5rway. 
Therefore, p determines the value of money paid out in the future. 



15.1.1 The Future Value of Money 

So for an n-year, m-payment annuity, the first payment of m dollars is truly worth 
m dollars. But the second payment a year later is worth only m/{l + p) dollars. 
Similarly, the third payment is worth m/(l + p^, and the n-th payment is worth 
only m/(l + p)"^^. The total value, V, of the annuity is equal to the sum of the 
payment values. This gives: 



{i+pY 



/ 1 y 

( ^— j— I (substitute j ::= i - 1) 



3=0 

-1 



Vx^ (substitute a; = ). (15.1) 

^ 1 + p 



The summation in (15.1) is a geometric sum that has a closed form, making the 
evaluation a lot easier, namely^. 



"^1 1 

^'•-T^^ ,15,2, 

1=0 

(The phrase "closed form" refers to a mathematical expression without any sum- 
mation or product notation.) 

Equation (15.2) was proved by induction in problem 6.2, but, as is often the 
case, the proof by induction gave no hint about how the formula was found in the 
first place. So weTl take this opportunity to explain where it comes from. The trick 
is to let S be the value of the sum and then observe what —xS is: 

S ^ 1 +x +x^ + ■■ ■ +a;"~i 

^V* O - — - /jt /-f^ /-vi3 _ _ _ /y'lT' 1 rytfT' 

iXj ti^ tJU Jb JU tXJ m 

Adding these two equations gives: 

S-xS^l-x"", 



so 

1 — X 

WeTl look further into this method of proof in a few weeks when we introduce 
generating functions in Chapter 16. 

^To make this equality liold for x = 0, we adopt the convention that O" ::= 1. 



15.1. THEVALUEOF AN ANNUITY 



309 



15.1.2 Closed Form for the Annuity Value 

So now we have a simple formula for V, the value of an annuity that pays m dollars 
at the start of each year for n years. 

1 — r" 

V^m (by (15.1) and (15.2)) (15.3) 

1 — a; 

^^^ i+p-ii/ii+p)r' (x = i/(i+p)). (15.4) 

p 

The formula (15.4) is much easier to use than a summation with dozens of terms. 
For example, what is the real value of a winning lottery ticket that pays $50, 000 
per year for 20 years? Plugging in to = $50, 000, n = 20, and p = 0.08 gives 
V $530, 180. So because payments are deferred, the million dollar lottery is 
really only worth about a half million dollars! This is a good trick for the lottery 
advertisers! 



15.1.3 Infinite Geometric Series 

The question we began with was whether you would prefer a million dollars today 
or $50, 000 a year for the rest of your life. Of course, this depends on how long you 
live, so optimistically assume that the second option is to receive $50, 000 a year 
forever. This sounds like infinite money! But we can compute the value of an 
annuity with an infinite number of payments by taking the limit of our geometric 
sum in (15.2) as n tends to infinity. 



Theorem 15.1.1. If\x\ < 1, then 



oo 



Proof. 



n-1 



Ex^ ::= lim > 
n — *oo — ^ 



i=0 
1 - t" 

= lim (by (15.2)) 

1 

~ \ -X 

The final line foUows from that fact that lim„^oo a;" — when |a;| < 1. 



310 



CHAPTER 15. SUMS & ASYMPTOTICS 



In our annuity problem, a:; = l/(l+p)<l, so Theorem 15.1.1 applies, and we 

get 



CSO 



V = m-^x^ (by (15.1)) 

j=0 



1 



1 -X 

l+p 



P 



(by Theorem 15.1.1) 



Plugging in TO = $50, 000 and p 0.08, the value, V, is only $675, 000. Amazingly 
a million dollars today is worth much more than $50, 000 paid every year forever! 
Then again, if we had a million dollars today in the bank earning 8% interest, we 
could take out and spend $80, 000 a year forever So on second thought, this answer 
really isn't so amazing. 

15.1.4 Problems 

Class Problems 
Problem 15.1. 

You've seen this neat trick for evaluating a geometric sum: 



S'=l + z + z^ + ... + z" 

zS = z + z"^ + . . . + z"- + z 

z 

1 - 



S=Z+Z^+ ,"+1 

S^zS=l- -"+1 



S 



1 - z 

Use the same approach to find a closed-form expression for this sum: 

T^lz + 2z'^ + 3z^ + ... + nz'' 

Homework Problems 
Problem 15.2. 

Is a Harvard degree really worth more than an MIT degree?! Let us say that a 
person with a Harvard degree starts with $40,000 and gets a $20,000 raise every 
year after graduation, whereas a person with an MIT degree starts with $30,000, 
but gets a 20% raise every year Assume inflation is a fixed 8% every year. That is, 
$1.08 a year from now is worth $1.00 today. 

(a) How much is a Harvard degree worth today if the holder will work for n years 
following graduation? 



(b) How much is an MIT degree worth in this case? 
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(c) If you plan to retire after twenty years, which degree would be worth more? 



Problem 15.3. 

Suppose you deposit $100 into your MIT Credit Union account today, $99 in one 
month from now, $98 in two months from now, and so on. Given that the interest 
rate is constantly 0.3% per month, how long will it take to save $5,000? 



Suppose you have a pile of books and you want to stack them on a table in some 
off-center way so the top book sticks out past books below it. How far past the 
edge of the table do you think you could get the top book to go without having the 
stack fall over? Could the top book stick out completely beyond the edge of table? 

Most people's first response to this question — sometimes also their second and 
third responses — is "No, the top book will never get completely past the edge of 
the table." But in fact, you can get the top book to stick out as far as you want: one 
booklength, two booklengths, any number of booklengths! 

15.2.1 Formalizing the Problem 

We'll approach this problem recursively. How far past the end of the table can we 
get one book to stick out? It won't tip as long as its center of mass is over the table, 
so we can get it to stick out half its length, as shown in Figure 15.1. 



Now suppose we have a stack of books that will stick out past the table edge 
without tipping over — call that a stable stack. Let's define the overhang of a stable 
stack to be the largest horizontal distance from the center of mass of the stack to 
the furthest edge of a book. If we place the center of mass of the stable stack at the 
edge of the table as in Figure 15.2, that's how far we can get a book in the stack to 
stick out past the edge. 



15.2 Book Stacking 




center of mass 
/ of book 



Figure 15.1: One book can overhang half a book length. 
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Figure 15.2: Overhanging the edge of the table. 



So we want a formula for the maximum possible overhang, i3„, achievable with 
a stack of n books. 

We've already observed that the overhang of one book is 1/2 a book length. 
That is. 




Now suppose we have a stable stack of n + 1 books with maximum overhang. 
If the overhang of the n books on top of the bottom book was not maximum, we 
could get a book to stick out further by replacing the top stack with a stack of n 
books with larger overhang. So the maximum overhang, i?„+i, of a stack of n + 1 
books is obtained by placing a maximum overhang stable stack of n books on top 
of the bottom book. And we get the biggest overhang for the stack of n + 1 books 
by placing the center of mass of the n books right over the edge of the bottom book 
as in Figure 15.3. 

So we know where to place the n + 1st book to get maximum overhang, and 
all we have to do is calculate what it is. The simplest way to do that is to let the 
center of mass of the top n books be the origin. That way the horizontal coordinate 
of the center of mass of the whole stack of n + 1 books will equal the increase 
in the overhang. But now the center of mass of the bottom book has horizontal 
coordinate 1 /2, so the horizontal coordinate of center of mass of the whole stack of 
n + 1 books is 

-71+ (1/2) -1 _ 1 

n+l ^ 2(n + 1)' 

In other words, 

i?„+i^i3„+ \ (15.5) 
2(n + 1) 

as shown in Figure 15.3. 
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top « books 



Figure 15.3: Additional overhang with n+1 books. 



Expanding equation (15.5), we have 

Bn+l = Bn-1 + h — — T 

2n 2(n + 1) 



1 1 

-Y- 

i=l 



2-2 2n 2(n+l) 

(15.6) 



The nth Harmonic number, Hn, is defined to be 
Definition 15.2.1. 

" 1 

z— 1 

So (15.6) means that 

The first few Harmonic numbers are easy to compute. For example, 



4 



l+^ + l + l = y|. The fact that H4 is greater than 2 has special significance; it 
implies that the total extension of a 4-book stack is greater than one full book! This 
is the situation shown in Figure 15.4. 

15.2.2 Evaluating the Sum — The Integral Method 

It would be nice to answer questions like, "How many books are needed to build a 
stack extending 100 book lengths beyond the table?" One approach to this question 
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Figure 15.4: Stack of four books with maximum overhang. 



would be to keep computing Harmonic numbers until we found one exceeding 
200. However, as we will see, this is not such a keen idea. 

Such questions would be settled if we could express in a closed form. Un- 
fortunately, no closed form is known, and probably none exists. As a second best, 
however, we can find closed forms for very good approximations to Hn using the 
Integral Method. The idea of the Integral Method is to bound terms of the sum 
above and below by simple functions as suggested in Figure 15.5. The integrals of 
these functions then bound the value of the sum above and below. 




1 /(x+ 1) 



Ol2 3 45678 

Figure 15.5: This figure illustrates the Integral Method for bounding a sum. The area 
under the "stairstep" curve over the interval [0,n] is equal to Hn = ^"=1 "^^^ 
function 1 /x is everywhere greater than or equal to the stairstep and so the integral ofl/x 
over this interval is an upper bound on the sum. Similarly, l/{x + 1) is everywhere less 
than or equal to the stairstep and so the integral of\/{x+\)isa lower bound on the sum. 



The Integral Method gives the following upper and lower bounds on the har- 
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monic number _ff„: 



iJ„ < 1+/ -da;=l + lnn (15.7) 

X 



1 



1 . 1 



Hr, > / dx^ - dx^lii(n+l). (15.8) 

7o 2; + 1 J I X 

These bounds imply that the harmonic number is around In n. 

But hi n grows — slowly — but without bound. That means we can get books to 
overhang any distance past the edge of the table by piling them high enough! For 
example, to build a stack extending three book lengths beyond the table, we need 
a number of books n so that > 6. By inequality (15.8), this means we want 

Hn > hi(n + 1) > 6, 

so n > — 1 books will work, that is, 403 books will be enough to get a three book 
overhang. Actual calculation of Hq shows that 227 books is the smallest number 
that will work. 



15.2.3 More about Harmonic Numbers 

In the preceding section, we showed that iJ„ is about In n. An even better approx- 
imation is known: 

H„ = In n + 7 + 



2n 12n2 120^4 

Here 7 is a value 0.577215664 . . . called Euler's constant, and e{n) is between and 
1 for all n. We will not prove this formula. 

Asymptotic Equality 

The shorthand iJ„ ^ In n is used to indicate that the leading term of i?„ is In n. 
More precisely: 

Definition 15.2.2. For functions /, 5 : M ^ M, we say / is asymptotically equal to g, 
in symbols, 

f{x) - g{x) 

iff 

lim f{x)/g{x) = 1. 

X — >CXD 

It's tempting to might write iJ„ ^ In n + 7 to indicate the two leading terms, 
but it is not really right. According to Definition 15.2.2, Hn ^ In n + c where c 
is any constant. The correct way to indicate that 7 is the second-largest term is 

Hn -hin^ 7. 
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The reason that the ^ notation is useful is that often we do not care about lower 
order terms. For example, if n = 100, then we can compute H{n) to great precision 
using only the two leading terms: 



15.2.4 Problems 

Class Problems 
Problem 15.4. 

An explorer is trying to reach the Holy Grail, which she believes is located in a 
desert shrine d days walk from the nearest oasis. In the desert heat, the explorer 
must drink continuously. She can carry at most 1 gallon of water, which is enough 
for 1 day. However, she is free to make multiple trips carrying up to a gallon each 
time to create water caches out in the desert. 

For example, if the shrine were 2/3 of a day's walk into the desert, then she 
could recover the Holy Grail after two days using the following strategy. She 
leaves the oasis with 1 gallon of water, travels 1/3 day into the desert, caches 1/3 
gallon, and then walks back to the oasis — arriving just as her water supply runs 
out. Then she picks up another gallon of water at the oasis, walks 1/3 day into the 
desert, tops off her water supply by taking the 1 /3 gallon in her cache, walks the 
remaining 1/3 day to the shrine, grabs the Holy Grail, and then walks for 2/3 of a 
day back to the oasis — again arriving with no water to spare. 

But what if the shrine were located farther away? 

(a) What is the most distant point that the explorer can reach and then return to 
the oasis if she takes a total of only 1 gallon from the oasis? 

(b) What is the most distant point the explorer can reach and still return to the 
oasis if she takes a total of only 2 gallons from the oasis? No proof is required; just 
do the best you can. 

(c) The explorer will travel using a recursive strategy to go far into the desert and 
back drawing a total of n gallons of water from the oasis. Her strategy is to build 
up a cache of n — 1 gallons, plus enough to get home, a certain fraction of a day's 
distance into the desert. On the last delivery to the cache, instead of returning 
home, she proceeds recursively with her n — 1 gallon strategy to go farther into the 
desert and return to the cache. At this point, tihe cache has just enough water left 
to get her home. 

Prove that with n gallons of water, this strategy will get her i?„/2 days into the 
desert and back, where Hn is the nth Harmonic number: 



\Hn - Inn - 7| < 



1 1 1 



1 

200' 



200 120000 120-1004 




Conclude that she can reach the shrine, however far it is from the oasis. 
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(d) Suppose that the shrine is d = 10 days walk into the desert. Use the asymp- 
totic approximation ~ In n to show that it will take more than a million years 
for the explorer to recover the Holy Grail. 



Problem 15.5. 

There is a niimber a such that converges iff p < a. What is the value of o? 

Prove it. 

Homework Problems 
Problem 15.6. 

There is a bug on the edge of a 1-meter rug. The bug wants to cross to the other 
side of the rug. It crawls at 1 cm per second. However, at the end of each second, 
a malicious first-grader named Mildred Anderson stretches the rug by 1 meter. As- 
sume that her action is instantaneous and the rug stretches uniformly. Thus, here's 
what happens in the first few seconds: 

• The bug walks 1 cm in the first second, so 99 cm remain ahead. 

• Mildred stretches the rug by 1 meter, which doubles its length. So now there 
are 2 cm behind the bug and 198 cm ahead. 

• The bug walks another 1 cm in the next second, leaving 3 cm behind and 197 
cm ahead. 

• Then Mildred strikes, stretching the rug from 2 meters to 3 meters. So there 
are now 3 • (3/2) = 4.5 cm behind the bug and 197 • (3/2) = 295.5 cm ahead. 

• The bug walks another 1 cm in the third second, and so on. 

Your job is to determine this poor bug's fate. 

(a) During second i, what fraction of the rug does the bug cross? 

(b) Over the first n seconds, what fraction of the rug does the bug cross alto- 
gether? Express your answer in terms of the Harmonic number 

(c) The known universe is thought to be about 3 • 10^° light years in diameter. 
How many imiverse diameters must the bug travel to get to the end of the rug? 

15.3 Finding Summation Formulas 

The Integral Method offers a way to derive formulas like those for the svm of 
consecutive integers, 

n 
i=l 
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or for the sum of squares, 

(2n + l)(n+ l)n 




(15.9) 



These equations appeared in Chapter 2 as equations (2.2) and (2.3) where they 
were proved using the Well-ordering Principle. But those proofs did not explain 
how someone figured out in the first place that these were the formulas to prove. 

Here's how the Integral Method leads to the sum-of-squares formula, for ex- 
ample. First, get a quick estimate of the sum: 

/ < V < / {x + 1)2 dx, 

Jo Jo 

so 

n 

nV3<^i2<(n+ 1)3/3 -1/3. (15.10) 

1=1 

and the upper and lower bounds (15.10) imply that 

n 

^z2^nV3. 

i=l 

To get an exact formula, we then guess the general form of the solution. Where we 
are uncertain, we can add parameters a,b,c, . . . . For example, we might make the 
guess: 

n 

^ = an^ + hn^ + cn + d. 

1=1 

If the guess is correct, then we can determine the parameters a, b, c, and d by 
plugging in a few values for n. Each such value gives a linear equation in a, b, 
c, and d. If we plug in enough values, we may get a linear system with a unique 
solution. Applying this method to our example gives: 

n = implies = d 

n — 1 implies l^a + b + c+d 

n = 2 implies 5 = 8a + 4fe + 2c + d 

n = 3 implies 14 = 27a + 9b + 3c+d. 

Solving this system gives the solution a = 1/3, b = 1/2, c = 1/6, rf = 0. Therefore, 
;/ our initial guess at the form of the solution was correct, then the summation is 
equal to n^/3 + r? 12 + n/6, which matches equation (15.9). 



E 
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The point is that if the desired formula turns out to be a pol5momial, then once 
you get an estimate of the degree of the pol5momial — by the Integral Method or 
any other way — all the coefficients of the pol5momial can be found automatically. 

Be careful! This method let's you discover formulas, but it doesn't guarantee 
they are right! After obtaining a formula by this method, it's important to go back 
and prove it using induction or some other method, because if the initial guess at 
the solution was not of the right form, then the resulting formula will be com- 
pletely wrong! 



15.3.1 Double Sums 

Sometimes we have to evaluate sums of sums, otherwise known as double sum- 
mations. This can be easy: evaluate the inner sum, replace it with a closed form, 
and then evaluate the outer sum which no longer has a summation inside it. For 
example. 



n=0 



1=0 / 

1 - a;"+i 

X 



^ ( y" — j (geometric sum formula (15.2)) 



\ — X 1 — X 

1 xYln=Q{^yy 



{l-y){l-x) l-X 

1 X 



{l-y){l-x) {l-xv){l-x) 

(1 - xy) - x{l ~ y) 
{l-xy){l-y){l-x) 
1 - x 

{l-xy){l~y){l~x) 
1 



(infinite geometric sum. Theorem 15.1.1) 
(infinite geometric sum. Theorem 15.1.1) 



(1 - xy){l - y)' 

When there's no obvious closed form for the inner sum, a special trick that is 
often useful is to try exchanging the order of summation. For example, suppose we 
want to compute the sum of the harmonic numbers 

n n k 

fc=i fc=ij=i 

For intuition about this sum, we can try the integral method: 

Hk ~ / In X rfa; w n In 7 

7 1 ^ 1 



nmn — n. 



k=l 
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Now let's look for an exact answer. If we think about the pairs {k, j) over which 
we are summing, they form a triangle: 





j 
1 


2 


3 


4 5 .. 


n 


k 1 


1 










2 


1 


1/2 








3 


1 


1/2 


1/3 






4 


1 


1/2 


1/3 


1/4 




n 


1 


1/2 






1/n 



The siunmation above is summing each row and then adding the row sums. In- 
stead, we can sum the columns and then add the column sums. Inspecting the 
table we see that this double sum can be written as 



fe=ii=i 



fe=i 



j=i k=j 

71 n 

= Ev.Ei 

j=i k=j 

n 

= E7("-^' + l) 



= E 



n-j + l 



En+1 s-^ j 

n ^ n 

= (-+i)E7-Ei 

= (n + 1)H„ - 11. 



(15.11) 



15.4 Stirling's Approximation 

The familiar factorial notation, n!, is an abbreviation for the product 



i=l 
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This is by far the most common product in discrete mathematics. In this section we 
describe a good closed-form estimate of nl called Stirling's Approximation. Unfor- 
tunately, all we can do is estimate: there is no closed form for nl — though proving 
so would take us beyond the scope of 6.042. 

15.4.1 Products to Sums 

A good way to handle a product is often to convert it into a sum by taking the 
logarithm. In the case of factorial, this gives 



We've not seen a summation containing a logarithm before! Fortunately, one tool 
that we used in evaluating sums is still applicable: the Integral Method. We can 
bound the terms of this sum with In x and ln(a: + 1) as shown in Figure 15.6. This 
gives bounds on ln(n!) as follows: 



The second line follows from the first by completing the integrations. The third 
line is obtained by exponentiating. 

So nl behaves something like the closed form formula {n/e)'^. A more careful 
analysis yields an unexpected closed form formula that is asymptotically exact: 

Lemma (Stirling's Formula). 



Stirling's Formula describes how nl behaves in the limit, but to use it effec- 
tively, we need to know how close it is to the limit for different values of n. That 
information is given by the bounding formulas: 

Fact (Stirling's Approximation). 



\n{nl) ==ln(l • 2 • 3 • • • (n - 1) • n) 

= In 1 + In 2 + In 3 H h ln(n - 1) + In n 



n 






(15.12) 
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ln(x) 



ln(x+ !)■ 



Figure 15.6: This figure illustrates the Integral Method for bounding the sum J27=i ^'^ 

Stirling's Approximation implies the asymptotic formula (15.12), since e^^ (i2n+i 
and e^/^^" both approach 1 as n grows large. These inequalities can be verified by 
induction, but the details are nasty. 

The bounds in Stirling's formula are very tight. For example, if n = 100, then 
Stirling's bounds are: 



The only difference between the upper bound and the lower bound is in the 
final term. In particular gi/i^oi ^ 1.00083299 and e^/^^°" w 1.00083368. As a 
result, the upper bound is no more than 1 + 10^^ times the lower bound. This is 
amazingly tight! Remember Stirling's formula; we will use it often. 

15.5 Asymptotic Notation 

Asymptotic notation is a shorthand used to give a quick measure of the behavior 
of a function /(n) as n grows large. 



The asymptotic notation, ^, of Definition 15.2.2 is a binary relation indicating that 
two functions grow at the same rate. There is also a binary relation indicating that 
one function grows at a significantly slower rate than another. Namely, 

Definition 15.5.1. For functions /, 5 : M ^ M, with g nonnegative, we say / is 




15.5.1 Little Oh 
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asymptotically smaller than g, in symbols, 

fix) = o{g{x)), 

iff 

lim f{x)/g{x) — 0. 

X — >co ' 

For example, lOOOx^-^ — o{x^), because lOOOx^-^/a;^ = 1000/a;*'-^ and since x^-^ 
goes to infinity with x and 1000 is constant, we have lim^^^ lOOOx^'^/a:^ = 0. This 
argument generalizes directly to yield 

Lemma 15.5.2. = o{x'')forall nonnegative constants a <b. 

Using the familiar fact that log x < x for all a; > 1, we can prove 

Lemma 15.5.3. log a; = o(x' ) for all e > and a; > 1 . 

Proof. Choose e > S > and let a; = z'' in the inequality log a; < a;. This implies 

log z < z^/S = o{z') by Lemma 15.5.2. (15.13) 

■ 

Corollary 15.5.4. x^ = o{a^)for any a, 6 e M zvith a> 1. 
Proof. From (15.13), 

logz < z^/S 

for all z > 1, (5 > 0. Hence 



_^ /gloga(6/loga) 



for all z such that 



^ ^(6/<51oga)^'> 



{b/Sloga)z^ < z. 



But choosing 5 < 1, we know = o(z), so this last inequality holds for all large 
enough z. ■ 

Lemma 15.5.3 and Corollary 15.5.4 can also be proved easily in several other 
ways, for example, using L'Hopital's Rule or the McLaurin Series for log x and e^. 
Proofs can be found in most calculus texts. 
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15.5.2 Big Oh 

Big Oh is the most frequently used asymptotic notation. It is used to give an upper 
bound on the growth of a function, such as the running time of an algorithm. 

Definition 15.5.5. Given nonnegative functions /, 5 : M ^ M, we say that 

/ = 0(5) 

iff 

limsupf{x)/g{x) < 00. 

X — >oo 

This definition^ makes it clear that 
Lemma 15.5.6. If f = o{g) or f ^ g, then f = 0{g). 

Proof, limf/g = or lim f /g = 1 implies limf/g < 00. ■ 

It is easy to see that the converse of Lemma 15.5.6 is not true. For example, 
2x = 0{x), but 2x ^ X and 2x ^ o(x). 

The usual formulation of Big Oh spells out the definition of lim sup without 
mentioning it. Namely, here is an equivalent definition: 

Definition 15.5.7. Given functions /, 5 : M M, we say that 

/ = 0(5) 

iff there exists a constant c > and an xq such that for all x > xq, \f{x) \ < cg{x). 

This definition is rather complicated, but the idea is simple: f{x) = 0{g{x)) 
means f{x) is less than or equal to g{x), except that we're willing to ignore a con- 
stant factor, namely, c, and to allow exceptions for small x, namely, x < xq. 

We observe. 

Lemma 15.5.8. If f — o{g), then it is not true that g — 0{f). 
Proof. 



lim ^ = 1 =-^00 

x^oo f{x) 1{x)/g{x) 



sog^O{f). 



^We can't simply use the limit as x — > 00 in the definition of 0(), because if f{x)/g{x) oscil- 
lates between, say, 3 and 5 as a; grows, then / = 0{g) because / < 5g, but lim^j^oo f(^)/g{^) 
does not exist. So instead of limit, we use the technical notion of lim sup. In this oscillating case, 
limsup^_^ f(x)/g(x) = 5. 

The precise definition of lim sup is 

\uas\r[)h(x) ::= lim \uby^^h{y), 
where "lub" abbreviates "least upper bound." 
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Proposition 15.5.9. lOOx^ = 0(a;2). 

Proof. Choose c = 100 and xq = 1. Then the proposition holds, since for all x > 1, 
1 100x2 1 < 100x2. ■ 

Proposition 15.5.10. x^ + lOOx + 10 = 0{x^). 

Proof, (x^ + lOOx + 10)/x^ = 1 + 100/x + lO/x^ and so its limit as x approaches 
infinity is 1 + 0+0 = 1. So in fact, x^ + lOOx+lO x^, and therefore x2 + 100x+ 10 = 
0(x2). Indeed, it's conversely true that x^ = 0{x'^ + lOOx + 10). ■ 

Proposition 15.5.10 generalizes to an arbitrary pol5momial: 

Proposition 15.5.11. For au ^ 0, akx^ + a^-xx^'^ + • • • + a\_x + ao = 0{x^). 

We'll omit the routine proof. 

Big Oh notation is especially useful when describing the running time of an al- 
gorithm. For example, the usual algorithm for multiplying n x n matrices requires 
proportional to -iv" operations in the worst case. This fact can be expressed con- 
cisely by saying that the running time is 0(n'^). So this asymptotic notation allows 
the speed of the algorithm to be discussed without reference to constant factors 
or lower-order terms that might be machine specific. In this case there is another, 
ingenious matrix multiplication procedure that requires Oiv?-^'^) operations. This 
procedure will therefore be much more efficient on large enough matrices. Un- 
fortunately, the 0{n}-^^Ya^QXdA\or\ multiplication procedure is almost never used 
because it happens to be less efficient than the usual Oirt") procedure on matrices 
of practical size. 

15.5.3 Theta 
Definition 15.5.12. 

/ = ©(.g) iff / = 0(5)and<?=0(/). 

The statement / = 0{g) can be paraphrased intuitively as "/ and g are equal to 
within a constant factor." 

The value of these notations is that they highlight growth rates and allow sup- 
pression of distracting factors and low-order terms. For example, if the running 
time of an algorithm is 

T(n) = lOn^ - 20^2 + 1, 

then 

T{n) = e{Tt-). 

In this case, we would say that T is of order r? or that T(n) grows cubically. 
Another such example is 

^23.-7 ^ (2.7x"3 + x9 - 86)^ _ ^^^3, ^ 
Vx 
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Just knowing that the running time of an algorithm is 0{n'^), for example, is 
useful, because if n doubles we can predict that the running time will by and largd^ 
increase by a factor of at most 8 for large n. In this way, Theta notation preserves in- 
formation about the scalability of an algorithm or system. Scalability is, of course, 
a big issue in the design of algorithms and systems. 

15.5.4 Pitfalls with Big Oh 

There is a long list of ways to make mistakes with Big Oh notation. This section 
presents some of the ways that Big Oh notation can lead to ruin and despair 

The Exponential Fiasco 

Sometimes relationships involving Big Oh are not so obvious. For example, one 
might guess that 4^ = 0(2^) since 4 is only a constant factor larger than 2. This 
reasoning is incorrect, however; actually 4^ grows much faster than 2^. 

Proposition 15.5.13. 4^ ^ 0(2^) 

Proof. 2^/4^ = 2^/(2^2^) = 1/2^. Hence, lim^^oo 2^/4^ = 0, so in fact 2^ = o(4=^). 
We observed earlier that this implies that 4^ ^ 0(2^). ■ 

Constant Confusion 

Every constant is 0(1). For example, 17 = 0{\). This is true because if we let 
/(x) = 17 and g{x) = 1, then there exists a c > and an xq such that \ f{x) \ < cg{x). 
In particular, we could choose c = 17 and xq — 1, since |17| < 17 • 1 for all a; > 1. 
We can construct a false theorem that exploits this fact. 

False Theorem 15.5.14. 

n 

4=1 

False proof. Define f{n) = Y^^=i « = l+ 2 + 3+ -- - + n. Since we have shown that 
every constant i is 0(1), /(n) = 0(1) + 0(1) + • • ■ + 0(1) = 0(n). ■ 

Of course in reality X]i"=i * — + l)/2 7^ 0{n). 

The error stems from confusion over what is meant in the statement i = 0(1). 
For any constant i e N it is true that i = 0(1). More precisely, if / is any constant 
function, then / = 0(1). But in this False Theorem, i is not constant but ranges 
over a set of values 0,1,. . . ,n that depends on n. 

And anjrway, we should not be adding 0(l)'s as though they were numbers. 
We never even defined what 0{g) means by itself; it should only be used in the 
context "/ = 0(g)" to describe a relation between functions / and g. 

■*Since ©(n^) only implies that the running time, T(n), is between cn"^ and d-n? for constants < 
c < d, the time T{2n) could regularly exceed T{n) by a factor as large as 8d/c. The factor is sure to be 
close to 8 for all large n only if T(n) ~ n^. 
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Lower Bound Blunder 

Sometimes people incorrectly use Big Oh in the context of a lower bound. For 
example, they might say, "The running time, T(n), is at least 0(n^)," when they 
probably mean something like "0(T(n)) = n^," or more properly, "n^ = 0(T(n))." 

Equality Blunder 

The notation / = 0{g) is too firmly entrenched to avoid, but the use of "=" is really 
regrettable. For example, if / = 0{g), it seems quite reasonable to write 0{g) = f. 
But doing so might tempt us to the following blunder: because 2n = 0{n), we can 
say 0(n) = 2n. But n = 0{n), so we conclude that n = 0{n) = 2n, and therefore 
n = 2n. To avoid such nonsense, we will never write "0{f) = g." 

15.5.5 Problems 

Practice Problems 
Problem 15.7. 

Let /(n) = n^. For each ftinction g{n) in the table below, indicate which of the 
indicated asymptotic relations hold. 



9{n) 


f = 0{g) 


f = o{g) 


9 = 0{f) 


9 = o{f) 


6 - 5n - 4n^ + 3n^ 










n"* log n 










(sin(7rn/2) + 2) 










^sin(7rn/2)+2 










logn! 





















Homework Problems 

Problem 15.8. (a) Prove that log x < x for all a; > 1 (requires elementary calculus). 

(b) Prove that the relation, R, on functions such that f R giii f = o{g) is a strict 
partial order. 

(c) Prove that f ^ g Hi f ~ g + hfor some function h = o{g). 



Problem 15.9. 

Indicate which of the following holds for each pair of functions {f{n),g{n)) in 
the table below. Assume fc > 1, e > 0, and c > 1 are constants. Pick the four 
table entries you consider to be the most challenging or interesting and justify 
your answers to these. 
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fin) 


gin) 


I = 0{g) 


f - o{g) 


9 = 0{f) 


9 - o{f) 


f = e(5) 


f-9 


2" 


2n/2 














Vn 


^sin n7r/2 














log(n!) 


log(n") 














n'' 


c" 














log'^ n 

















Problem 15.10. 

Let /, g be nonnegative real-valued functions such that liniaj^oo f{x) = oo and 

(a) Give an example of /, g such that N0T(2-^ ^ 2^). 

(b) Prove that log / ^ log g. 

(c) Use Stirling's formula to prove that in fact 

log(n!) ~ rtlogn 

Class Problems 
Problem 15.11. 

Give an elementary proof (without appealing to Stirling's formula) that log (71!) = 
0(nlogn). 



Problem 15.12. 

Recall that for functions /, g on N, / = 0{g) iff 

3c e N3no e NVn > no c ■ g{n) > \f{n)\ . (15.14) 

For each pair of functions below, determine whether / = 0{g) and whether 
g — 0{f). In cases where one function is 0() of the other, indicate the smallest 
nonegative integer, c, and for that smallest c, the smallest corresponding nonegative 
integer uq ensuring that condition (15.14) applies. 

(a) f{n) — n^,g{n) = 3n. 

f = 0{g) YES NO lfYES,c = ,no = 

g = 0{f) YES NO lfYES,c = ,no = 

(b) f{n) = (3n - 7)/(n + A),g{n) = 4 

f = 0{g) YES NO lfYES,c = ,no= 

g = 0{f) YES NO lfYES,c = ,no = 
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(c) /(n) = 1 + (nsin(n7r/2))2,.g(n) = 3n 
f = 0{g) YES NO If yes, c 

g=0{f) YES NO If yes, c 

Problem 15.13. 

False Claim. 

2" = 0(1). (15.15) 

Explain why the claim is false. Then identify and explain the mistake in the 
following bogus proof. 

Bogus proof. The proof by induction on n where the induction hjrpothesis, P{n), is 
the assertion (15.15). 

base case: P(0) holds trivially. 

inductive step: We may assume P{n), so there is a constant c > such that 
2" < c • 1 . Therefore, 

2"+i = 2-2"<(2c)-l, 

which implies that 2"+^ = 0(1). That is, P(n + 1) holds, which completes the proof 
of the inductive step. 

We conclude by induction that 2" = 0(1) for all n. That is, the exponential 
fimction is boimded by a constant. 



Problem 15.14. 

(a) Define a function f{n) such that / = 6(n^) and N0T(/ ~ n^). 

(b) Define a function g{n) such that g = 0{n^), g ^ Q{n^) and g ^ o{n^). 



no = 
no = 
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